Computer based structured storage systems, such as computer file systems and database systems, have been remarkably successful at providing users with quick and facile access to enormous amounts of data. The importance of these structured storage systems in today's commerce is difficult to exaggerate. For example, structured storage systems have allowed businesses to generate and maintain enormous stores of persistent data that the company can modify and update over the course of years. For many companies, this persistent data is a valuable capital asset that is employed each day to perform the company's core operations. The data can be, for example, computer files (e.g., source code, wordprocessing documents, etc.), database records and information (e.g., information on employees, customers, and/or products), and/or Web pages.
A typical computer based structured storage system includes a central server, such as a file system server or a database system server, that provides centralized control over the structured store of data. The structured store of data is the information that is being maintained by the system, such as the information in the files and the directories of the file system or within the rows and columns of the tables of the database system. The central server provides system services to a plurality of interconnected network client nodes, and each of the client nodes employs the central server to access and manipulate the structured store of data. Accordingly, the central server provides a nucleus for the structured storage system and maintains central control over the system and the data stored therein.
Although such server based systems have worked generally well, problems arise from relying on centralized control of the structured data store. For example, the operation of the structured storage system is dependent upon the proper functioning of the central server. Any failure of the server to maintain proper operation, such as a power failure, hardware failure, or other such system failure, will disable the entire structured storage system and prevent users from accessing the data store. Additionally, a flood of client service requests issued from the individual network nodes can overload the server process and slow down or crash the system. Accordingly, reliance on centralized control of the structured storage system can result in slow operation during periods of heavy use, as well as result in system failures due to overloading the central server.
An additional problem with a client-server network system is that it provides a static operating environment that is set for optimal performance at a certain level of network activity. Consequently, the network fails to exploit available resources to improve system performance. In particular, as the system activity rises above or drops below the expected level of network activity, the static operating environment lacks any ability to reconfigure dynamically the allocation of network resources to one providing better performance for the present level of activity.
Technology has been developed to improve the reliability and operation of these centralized structured storage network systems. This technology has been mostly directed to the development of reliable database and file systems, and has generally involved one of two methods: (1) static mapping of the data to one or more servers, or (2) storing the data in a globally shared data repository, such as a shared disk.
Systems using the first method distribute portions of the structured store of persistent data statically across a plurality of servers. Each of the servers maintains a portion of the structured store of data, as well as optionally maintaining an associated portion of a directory structure that describes the portions of the data stored within that particular server. These systems guard against a loss of data by distributing the storage of data statically across a plurality of servers such that the failure of any one server will result in a loss of only a portion of the overall data. Other developments in clustered database technology provide for replicating portions of the structured store of data, and storing the replicated portions statically across a plurality of servers. Accordingly, these systems go further in guarding against the loss of data by providing static redundancy within the structured storage system. However, although known clustered database technology can provide more fault tolerant operation in that it guards against data loss, the known systems still rely on static allocation of the data across various servers. Since data is not dynamically allocated between servers: (1) system resources are not allocated based on system usage which results in under utilization of those resources; (2) scaleable performance is limited because new servers must be provided whenever the dataset grows or whenever one particular server cannot service requests made to its portion of the dataset; and (3) such static allocation still requires at least one of servers storing the information to survive in order to preserve the data.
Systems using the second method store the structured data in central data repository, such as a shared disk. Each node in the system continually updates the central data repository with its portion of the structured store. For example, in a database system, each node exports tables it is currently using to the data store. While this method exports the problems of load balancing to the central data repository, it suffers from two main drawbacks. First, throughput is lowered because of increased overhead associated with ensuring coherency of the centralized data store. Second, locking is inefficient because entire pages are locked when a node accesses any portion of a page. As a result, nodes may experience contention for memory even when no true conflict exists.